Pose Estimation Research Articles

VirTeach: mmWave Radar Point-Cloud-Based Pose Estimation With Virtual Data as a Teacher

VirTeach: mmWave Radar Point-Cloud-Based Pose Estimation With Virtual Data as a Teacher

Self-supervised monocular visual odometry based on cross-correlation

Visual odometry constitutes a critical component in enabling autonomous navigation. However, the existing methods are limited by the feature extraction and matching accuracy, and cannot show good real-time performance while combining accuracy and robustness. In this paper, we propose a novel monocular visual odometry framework based on cross-correlation. The framework starts with a parameter-sharing Siamese network to build feature extractors that can simultaneously process multiple images as inputs. Moreover, we design cross-correlation modules and define a cross-correlation matrix to describe the strength of correlation between different parts of the input feature maps, reflecting the rotational and translational transformations of the input images. Furthermore, a novel loss function is introduced to impose constraints on the network. Additionally, a fully convolutional network is designed for pose estimation, computing poses alterations from the structure of the cross-correlation matrix. Channel attention and spatial attention mechanisms are introduced to improve the performance. More importantly, our method innovatively uses time intervals as labels, enables self-supervised training, and relies only on a monocular camera. Experimental results on the KITTI visual odometry dataset and the Oxford Robotcar Dataset show that our method produces competitive performance, demonstrating the superiority of the proposed method.

BACK-to-MOVE: Machine learning and computer vision model automating clinical classification of non-specific low back pain for personalised management.

Low back pain (LBP) is a major global disability contributor with profound health and socio-economic implications. The predominant form is non-specific LBP (NSLBP), lacking treatable pathology. Active physical interventions tailored to individual needs and capabilities are crucial for its management. However, the intricate nature of NSLBP and complexity of clinical classification systems necessitating extensive clinical training, hinder customised treatment access. Recent advancements in machine learning and computer vision demonstrate promise in characterising NSLBP altered movement patters through wearable sensors and optical motion capture. This study aimed to develop and evaluate a machine learning model (i.e., 'BACK-to-MOVE') for NSLBP classification trained with expert clinical classification, spinal motion data from a standard video alongside patient-reported outcome measures (PROMs). Synchronised video and three-dimensional (3D) motion data was collected during forward spinal flexion from 83 NSLBP patients. Two physiotherapists independently classified them as motor control impairment (MCI) or movement impairment (MI), with conflicts resolved by a third expert. The Convolutional Neural Networks (CNNs) architecture, HigherHRNet, was chosen for effective pose estimation from video data. The model was validated against 3D motion data (subset of 62) and trained on the freely available MS-COCO dataset for feature extraction. The Back-to-Move classifier underwent fine-tuning through feed-forward neural networks using labelled examples from the training dataset. Evaluation utilised 5-fold cross-validation to assess accuracy, specificity, sensitivity, and F1 measure. Pose estimation's Mean Square Error of 0.35 degrees against 3D motion data demonstrated strong criterion validity. Back-to-Move proficiently differentiated MI and MCI classes, yielding 93.98% accuracy, 96.49% sensitivity (MI detection), 88.46% specificity (MCI detection), and an F1 measure of .957. Incorporating PROMs curtailed classifier performance (accuracy: 68.67%, sensitivity: 91.23%, specificity: 18.52%, F1: .800). This study is the first to demonstrate automated clinical classification of NSLBP using computer vision and machine learning with standard video data, achieving accuracy comparable to expert consensus. Automated classification of NSLBP based on altered movement patters video-recorded during routine clinical examination could expedite personalised NSLBP rehabilitation management, circumventing existing healthcare constraints. This advancement holds significant promise for patients and healthcare services alike.

Toward Robotic Nuclear Decommissioning: Deep Learning-Based Object Classification and Pose Estimation from Partial-View Scans

This study introduces a novel framework for the robotic decommissioning of nuclear facilities, that focuses on object classification and six degrees of freedom pose estimation from partial-view three-dimensional (3-D) scan data. Addressing the challenge of precise robotic manipulation in environments where acquiring full-scan data is impractical, this framework leverages a deep neural network for initial pose estimation, subsequently refined by a modified iterative closest point algorithm. Our method demonstrates high accuracy in identifying scanned objects and estimating their poses from partial-view scans, validated through experiments with 3-D printed mock-ups. This advancement highlights the potential for significantly enhancing robotic automation in nuclear decommissioning and related fields.

Comparing novel smartphone pose estimation frameworks with the Kinect V2 for knee tracking during athletic stress tests.

To compare the accuracy of the Microsoft Kinect V2 with novel pose estimation frameworks, in assessing knee kinematics during athletic stress tests, for fast and portable risk assessment of anterior cruciate ligament (ACL) injury. We captured 254 varsity athletes, using the Kinect V2 and a smartphone application utilizing Google's MediaPipe framework. The devices were placed as close as possible and used to capture a person, facing the cameras, performing one of three athletic stress tests at a distance of 2.5ms. Custom software translated the results from both frameworks to the same format. We then extracted relevant knee angles at key moments of the jump and compared them, using the Kinect V2 as the ground truth. The results show relatively small angle differences between the two solutions in the coronal plane and moderate angle differences on the sagittal plane. Overall, the MediaPipe framework results seem to underestimate both knee valgus angles and knee sagittal angles compared to the Kinect V2. This preliminary study demonstrates the potential for Google's MediaPipe framework to be used for calculating lower limb kinematics during athletic stress test motions, which can run on most modern smartphones, as it produces similar results to the Kinect V2. A smartphone application similar to the one developed could potentially be used for low cost and widespread ACL injury prevention.

Research on Human Posture Estimation Algorithm Based on YOLO-Pose

In response to the numerous challenges faced by traditional human pose recognition methods in practical applications, such as dense targets, severe edge occlusion, limited application scenarios, complex backgrounds, and poor recognition accuracy when targets are occluded, this paper proposes a YOLO-Pose algorithm for human pose estimation. The specific improvements are divided into four parts. Firstly, in the Backbone section of the YOLO-Pose model, lightweight GhostNet modules are introduced to reduce the model’s parameter count and computational requirements, making it suitable for deployment on unmanned aerial vehicles (UAVs). Secondly, the ACmix attention mechanism is integrated into the Neck section to improve detection speed during object judgment and localization. Furthermore, in the Head section, key points are optimized using coordinate attention mechanisms, significantly enhancing key point localization accuracy. Lastly, the paper improves the loss function and confidence function to enhance the model’s robustness. Experimental results demonstrate that the improved model achieves a 95.58% improvement in mAP50 and a 69.54% improvement in mAP50-95 compared to the original model, with a reduction of 14.6 M parameters. The model achieves a detection speed of 19.9 ms per image, optimized by 30% and 39.5% compared to the original model. Comparisons with other algorithms such as Faster R-CNN, SSD, YOLOv4, and YOLOv7 demonstrate varying degrees of performance improvement.

M-GCLO: Multiple Ground Constrained LiDAR Odometry

Abstract. Accurate LiDAR odometry results contribute directly to high-quality point cloud maps. However, traditional LiDAR odometry methods drift easily upward, leading to inaccuracies and inconsistencies in the point cloud maps. Considering abundant and reliable ground points in the Mobile Mapping System(MMS), ground points can be extracted, and constraints can be built to eliminate pose drifts. However, existing LiDAR-based odometry methods either do not use ground point cloud constraints or consider the ground plane as an infinite plane (i.e., single ground constraint), making pose estimation prone to errors. Therefore, this paper is dedicated to developing a Multiple Ground Constrained LiDAR Odometry(M-GCLO) method, which extracts multiple grounds and optimizes those plane parameters for better accuracy and robustness. M-GCLO includes three modules. Firstly, the original point clouds will be classified into the ground and non-ground points. Ground points are voxelized, and multiple ground planes are extracted, parameterized, and optimized to constrain the pose errors. All the non-ground point clouds are used for point-to-distribution matching by maintaining an NDT voxel map. Secondly, a novel method for weighting the residuals is proposed by considering the uncertainties of each point in a scan. Finally, the jacobians and residuals are given along with the weightings for estimating LiDAR states. Experimental results in KITTI and M2DGR datasets show that M-GCLO outperforms state-of-the-art LiDAR odometry methods in large-scale outdoor and indoor scenarios.

Realtime 3D reconstruction algorithm based on improved AKAZE and multi-layer feature detection network

Simultaneous localization and mapping (SLAM) technology aims to solve the problem of how to determine the trajectory of robots by observing the environment and construct sparse spatial models. And the 3D reconstruction technology can further build dense spatial models based on the pose information. This paper proposes a dense 3D reconstruction algorithm based on improved accelerated-KAZE features and a multi-layer feature detection network. A multi-layer feature detection neural network is constructed to replace the bag-of-words model and optimize the loopback results, which improves the effect of pose estimation. The optimized algorithm improves the mapping effect and construct clear and robust indoor complex models. The pose estimation results show that the optimized algorithm improves the accuracy by 30% compared with ORB-SLAM2. Compared with the mapping effect of Kintinuous, ElasticFusion, ORB-SLAM2 and BundleFusion, the optimized models own the advantages of clearer details, richer information and no split layer.

Integration of nonlinear observer and unscented Kalman filter for pose estimation in autonomous truck–trailer and container truck

Integration of nonlinear observer and unscented Kalman filter for pose estimation in autonomous truck–trailer and container truck

Decoding Posture: Distinguishing Between Good and Bad Body Alignment

On the grounds of artificial intelligence sensors, the research focuses on achieving precise detection of intelligent posture to minimize the risk of injuries resulting from incorrect body positioning and to mitigate the negative impact on both performance and physical well-being. Utilizing an attitude analysis and recognition system that relies on the nertial measurement unit enables not only the measurement of human body motion information but also the acquisition of data pertaining to motion characteristics and body movement states through the examination of posture data. A proposition is presented for intelligent posture training that is centered on accuracy and real-time feedback. Furthermore, it is elucidated that the accuracy of posture recognition through machine learning is significantly influenced by variations in user BMI. This paper will introduce the most recent advancements in posture recognition techniques, providing an overview of the various methodologies and algorithms that have emerged in recent years. Additionally, it delves into enhanced approaches such as stacked hourglass networks, multi-stage pose estimation networks, convolutional pose machines, and high-resolution nets. The study thoroughly dissects and consolidates the general procedures and datasets involved in posture recognition, comparing several enhanced CNN methodologies alongside three principal recognition techniques. Moreover, it delves into the utilization of advanced neural networks in posture recognition, including but not limited to transfer learning, ensemble learning, graph neural networks, and explainable deep neural networks. Researchers have observed that CNN has made significant strides in the realm of posture recognition and has garnered favor, yet emphasize the necessity for further exploration into areas such as feature extraction and information fusion.

Implementation of hand‐object pose estimation using SSD and YOLOV5 model for object grasping by SCARA robot

AbstractEnforcement of advanced deep learning methods in hand‐object pose estimation is an imperative method for grasping the objects safely during the human–robot collaborative tasks. The position and orientation of a hand‐object from a two‐dimensional image is still a crucial problem under various circumstances like occlusion, critical lighting, and salient region detection and blur images. In this paper, the proposed method uses an enhanced MobileNetV3 with single shot detection (SSD) and YOLOv5 to ensure the improvement in accuracy and without compromising the latency in the detection of hand‐object pose and its orientation. To overcome the limitations of higher computation cost, latency and accuracy, the Network Architecture Search and NetAdapt Algorithm is used in MobileNetV3 that perform the network search for parameter tuning and adaptive learning for multiscale feature extraction and anchor box offset adjustment due to auto‐variance of weight in the level of each layers. The squeeze‐and‐excitation block reduces the computation and latency of the model. Hard‐swish activation function and feature pyramid networks are used to prevent over fitting the data and stabilizing the training. Based on the comparative analysis of MobileNetV3 with its predecessor and YOLOV5 are carried out, the obtained results are 92.8% and 89.7% of precision value, recall value of 93.1% and 90.2%, mAP value of 93.3% and 89.2%, respectively. The proposed methods ensure better grasping for robots by providing the pose estimation and orientation of hand‐objects with tolerance of −1.9 to 2.15 mm along x, −1.55 to 2.21 mm along y, −0.833 to 1.51 mm along z axis and −0.233° to 0.273° along z‐axis.

3D-MuPPET: 3D Multi-Pigeon Pose Estimation and Tracking

AbstractMarkerless methods for animal posture tracking have been rapidly developing recently, but frameworks and benchmarks for tracking large animal groups in 3D are still lacking. To overcome this gap in the literature, we present 3D-MuPPET, a framework to estimate and track 3D poses of up to 10 pigeons at interactive speed using multiple camera views. We train a pose estimator to infer 2D keypoints and bounding boxes of multiple pigeons, then triangulate the keypoints to 3D. For identity matching of individuals in all views, we first dynamically match 2D detections to global identities in the first frame, then use a 2D tracker to maintain IDs across views in subsequent frames. We achieve comparable accuracy to a state of the art 3D pose estimator in terms of median error and Percentage of Correct Keypoints. Additionally, we benchmark the inference speed of 3D-MuPPET, with up to 9.45 fps in 2D and 1.89 fps in 3D, and perform quantitative tracking evaluation, which yields encouraging results. Finally, we showcase two novel applications for 3D-MuPPET. First, we train a model with data of single pigeons and achieve comparable results in 2D and 3D posture estimation for up to 5 pigeons. Second, we show that 3D-MuPPET also works in outdoors without additional annotations from natural environments. Both use cases simplify the domain shift to new species and environments, largely reducing annotation effort needed for 3D posture tracking. To the best of our knowledge we are the first to present a framework for 2D/3D animal posture and trajectory tracking that works in both indoor and outdoor environments for up to 10 individuals. We hope that the framework can open up new opportunities in studying animal collective behaviour and encourages further developments in 3D multi-animal posture tracking.

LEAPSE: Learning Environment Affordances for 3D Human Pose and Shape Estimation.

We live in a 3D world where people interact with each other in the environment. Learning 3D posed humans therefore requires us to perceive and interpret these interactions. This paper proposes LEAPSE, a novel method that learns salient instance affordances for estimating a posed body from a single RGB image in a non-parametric manner. Existing methods mostly ignore the environment and estimate the human body independently from the surroundings. We capture the influences of non-contact and contact instances on a posed body as an adequate representation of the "environment affordances". The proposed method learns the global relationships between 3D joints, body mesh vertices, and salient instances as environment affordances on the human body. LEAPSE achieved state-of-the-art results on the 3DPW dataset with many affordance instances, and also demonstrated excellent performance on Human3.6M dataset. We further demonstrate the benefit of our method by showing that the performance of existing weak models can be significantly improved when combined with our environment affordance module.

Improved markerless gait kinematics measurement using a biomechanically-aware algorithm with subject-specific geometric modeling

Despite the advancements in developing markerless gait analysis systems, they still demonstrate lower accuracy compared to gold-standard systems. Hence, in this research, a novel approach is presented to improve the lower limb kinematics accuracy in markerless gait analysis. This approach refines the 3D lower-limb skeletons obtained by AI-based pose estimation algorithms in a subject-specific geometric manner, preserves skeleton links’ length, benefits from gait phases information that adds biomechanical awareness to the algorithm, and utilizes an embedded trajectory smoothing. Validation of the proposed method shows that it reduces 12.6–43.5 % of root mean square error (RMSE) and significantly improves kinematic curves’ similarity to the gold-standard ones. Results also prove the feasibility of more accurate lower limb kinematics calculation using a single (2.02–7.57° RMSE) or dual RGB-D camera (1.66–7.25° RMSE). Development of such algorithms could result in requirement of fewer cameras that deliver comparable or even superior measurement accuracy compared to multi-camera approaches.

Integration of Eye-Tracking and Object Detection in a Deep Learning System for Quality Inspection Analysis

Abstract During quality inspection in manufacturing, the gaze of a worker provides pivotal information for identifying surface defects of a product. However, it is challenging to digitize the gaze information of workers in a dynamic environment where the positions and postures of the products and workers are not fixed. A robust, deep learning-based system, ISGOD (Integrated System with worker's Gaze and Object Detection), is proposed, which analyzes data to determine which part of the object is observed by integrating object detection and eye-tracking information in dynamic environments. The ISGOD employs a 6D pose estimation algorithm for object detection, considering the location, orientation, and rotation of the object. Eye-tracking data were obtained from Tobii Glasses, which enable real-time video transmission and eye-movement tracking. A latency reduction method is proposed to overcome the time delays between object detection and eye-tracking information. Three evaluation indices, namely, gaze score, accuracy score, and concentration index are suggested for comprehensive analysis. Two experiments were conducted: a robustness test to confirm the suitability for real-time object detection and eye-tracking, and a trend test to analyze the difference in gaze movement between experts and novices. In the future, the proposed method and system can transfer the expertise of experts to enhance defect detection efficiency significantly.

Pedestrian Pose Recognition Based on Frequency-Modulated Continuous-Wave Radar with Meta-Learning.

With the continuous advancement of autonomous driving and monitoring technologies, there is increasing attention on non-intrusive target monitoring and recognition. This paper proposes an ArcFace SE-attention model-agnostic meta-learning approach (AS-MAML) by integrating attention mechanisms into residual networks for pedestrian gait recognition using frequency-modulated continuous-wave (FMCW) millimeter-wave radar through meta-learning. We enhance the feature extraction capability of the base network using channel attention mechanisms and integrate the additive angular margin loss function (ArcFace loss) into the inner loop of MAML to constrain inner loop optimization and improve radar discrimination. Then, this network is used to classify small-sample micro-Doppler images obtained from millimeter-wave radar as the data source for pose recognition. Experimental tests were conducted on pose estimation and image classification tasks. The results demonstrate significant detection and recognition performance, with an accuracy of 94.5%, accompanied by a 95% confidence interval. Additionally, on the open-source dataset DIAT-μRadHAR, which is specially processed to increase classification difficulty, the network achieves a classification accuracy of 85.9%.

Estimating the Thumb Rotation Angle by Using a Tablet Device With a Posture Estimation Artificial Intelligence Model.

MediaPipe Hand (MediaPipe) is an artificial intelligence (AI)-based pose estimation library. In this study, MediaPipe was combined with four machine learning (ML) models to estimate the rotation angle of the thumb. Videos of the right hands of 15 healthy volunteers were recorded and processed into 9000 images. The rotation angle of the thumb (defined as angle θ from the palmar plane, which is defined as 0°) was measured using an angle measuring device, expressed in a radiansystem. Angle θ was then estimated by the ML model by using parameters calculated from the hand coordinates detected by MediaPipe. The linear regression model showed a root mean square error (RMSE) of 12.23, a mean absolute error (MAE) of 9.9, and a correlation coefficient of 0.91. The ElasticNet model showed an RMSE of 12.23, an MAE of 9.95, and a correlation coefficient of 0.91; the support vector machine (SVM) model showed an RMSE of 4.7, an MAE of 2.5, and a correlation coefficient of 0.99. The LightGBM model achieved high values: an RMSE of 4.58, an MAE of 2.62, and a correlation coefficient of 0.99. Based on these findings, we concluded that the thumb rotation angle can be estimated with high accuracy by combining MediaPipe and ML.

Automatic loading method for robot guided by 3D vision for scattered and stacked shoe soles

This paper primarily introduces a method for the automated feeding of scattered stacked shoe soles using a 3D visual-guided robot. Initially, addressing issues like slow speed in pose estimation and poor robustness during the robot sorting and feeding process, we introduce an enhanced pose estimation algorithm. This algorithm combines the improved Super 4-Point Congruent Sets (Super 4PCS) with the Truncated Least Squares Semidefinite Relaxation (TEASER) algorithm, significantly boosting the speed and robustness of pose estimation during sole sorting, and achieving precise target pose estimation. Building upon this foundation, we present a sorting strategy for disordered stacked shoe soles. This strategy integrates spatial positional information of each sole, employing multi-objective decision-making and recognition algorithms to determine optimal grasping targets. Finally, an experimental platform for automated sole feeding is designed to validate the proposed method. The experimental results indicate that the pose estimation method proposed in this paper achieves an average distance error of 2.04 mm and an average angular error of 2.72°, with the robot’s average success rate in grasping reaching 97.08%. Moreover, the average processing time of the vision algorithm is only 1.34 s, demonstrating good efficiency, precision, and robustness. This method effectively meets the automated feeding needs of scattered and stacked shoe soles in the actual production processes of shoemaking enterprises.

3D hand pose and mesh estimation via a generic Topology-aware Transformer model

In Human-Robot Interaction (HRI), accurate 3D hand pose and mesh estimation hold critical importance. However, inferring reasonable and accurate poses in severe self-occlusion and high self-similarity remains an inherent challenge. In order to alleviate the ambiguity caused by invisible and similar joints during HRI, we propose a new Topology-aware Transformer network named HandGCNFormer with depth image as input, incorporating prior knowledge of hand kinematic topology into the network while modeling long-range contextual information. Specifically, we propose a novel Graphformer decoder with an additional Node-offset Graph Convolutional layer (NoffGConv). The Graphformer decoder optimizes the synergy between the Transformer and GCN, capturing long-range dependencies and local topological connections between joints. On top of that, we replace the standard MLP prediction head with a novel Topology-aware head to better exploit local topological constraints for more reasonable and accurate poses. Our method achieves state-of-the-art 3D hand pose estimation performance on four challenging datasets, including Hands2017, NYU, ICVL, and MSRA. To further demonstrate the effectiveness and scalability of our proposed Graphformer Decoder and Topology aware head, we extend our framework to HandGCNFormer-Mesh for the 3D hand mesh estimation task. The extended framework efficiently integrates a shape regressor with the original Graphformer Decoder and Topology aware head, producing Mano parameters. The results on the HO-3D dataset, which contains various and challenging occlusions, show that our HandGCNFormer-Mesh achieves competitive results compared to previous state-of-the-art 3D hand mesh estimation methods.

Estimation of Shoulder Joint Rotation Angle Using Tablet Device and Pose Estimation Artificial Intelligence Model.

Traditionally, angle measurements have been performed using a goniometer, but the complex motion of shoulder movement has made these measurements intricate. The angle of rotation of the shoulder is particularly difficult to measure from an upright position because of the complicated base and moving axes. In this study, we attempted to estimate the shoulder joint internal/external rotation angle using the combination of pose estimation artificial intelligence (AI) and a machine learning model. Videos of the right shoulder of 10 healthy volunteers (10 males, mean age 37.7 years, mean height 168.3 cm, mean weight 72.7 kg, mean BMI 25.6) were recorded and processed into 10,608 images. Parameters were created using the coordinates measured from the posture estimation AI, and these were used to train the machine learning model. The measured values from the smartphone's angle device were used as the true values to create a machine learning model. When measuring the parameters at each angle, we compared the performance of the machine learning model using both linear regression and Light GBM. When the pose estimation AI was trained using linear regression, a correlation coefficient of 0.971 was achieved, with a mean absolute error (MAE) of 5.778. When trained with Light GBM, the correlation coefficient was 0.999 and the MAE was 0.945. This method enables the estimation of internal and external rotation angles from a direct-facing position. This approach is considered to be valuable for analyzing motor movements during sports and rehabilitation.